Estimating future grid job costs by classifying grid jobs and storing results of processing grid job microcosms

ABSTRACT

A method, system, and program for estimating future grid job costs by classifying grid jobs and storing results of processing grid job microcosms are provided. In general, a client side agent estimates future grid job costs by comparing a current grid job of a particular classification with a history of stored costs for other grid jobs of that customer of that particular classification. In particular, the grid client agent for a client system enabled to submit grid jobs to a grid provider that facilitates a grid environment, calculates a ratio of an application based metric to a grid provider metric for processing a particular grid job. Then, the grid client agent creates a table with an entry comparing the application based metric to a cost per grid provider metric for the grid provider based on the calculated ratio. Next, the grid client agent stores the table with the entry. Then, responsive to detecting a next grid job, the grid client agent estimates a cost for the grid provider to process the next grid job based on a particular number of application based metric operations required for the next grid job, translated by the ratio into the grid provider metric and multiplied by the cost per grid provider metric.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is related to the following co-pendingapplications, hereby incorporated herein by reference:

-   -   (1) U.S. patent application Ser. No. ______ (Attorney Docket No.        AUS920031044US1); and    -   (2) U.S. patent application Ser. No. ______ (Attorney Docket No.        AUS920040050US1).

BACKGROUND OF THE INVENTION

1. Technical Field

The present invention relates in general to improved grid computing andin particular to efficient client-side estimation of future grid jobcosts. Still more particularly, the present invention relates tocustomer estimation of future grid job costs by comparing a current gridjob of a particular classification with a history of stored costs forother grid jobs of that customer of that particular classification.

2. Description of the Related Art

Ever since the first connection was made between two computer systems,new ways of transferring data, resources, and other information betweentwo computer systems via a connection continue to develop. In typicalnetwork architectures, when two computer systems are exchanging data viaa connection, one of the computer systems is considered a client sendingrequests and the other is considered a server processing the requestsand returning results. In an effort to increase the speed at whichrequests are handled, server systems continue to expand in size andspeed. Further, in an effort to handle peak periods when multiplerequests are arriving every second, server systems are often joinedtogether as a group and requests are distributed among the groupedservers. Multiple methods of grouping servers have developed such asclustering, multi-system shared data (sysplex) environments, andenterprise systems. With a cluster of servers, one server is typicallydesignated to manage distribution of incoming requests and outgoingresponses. The other servers typically operate in parallel to handle thedistributed requests from clients. Thus, one of multiple servers in acluster may service a client request without the client detecting that acluster of servers is processing the request.

Typically, servers or groups of servers operate on a particular networkplatform, such as Unix or some variation of Unix, and provide a hostingenvironment for running applications. Each network platform may providefunctions ranging from database integration, clustering services, andsecurity to workload management and problem determination. Each networkplatform typically offers different implementations, semantic behaviors,and application programming interfaces (APIs).

Merely grouping servers together to expand processing power, however, isa limited method of improving efficiency of response times in a network.Thus, increasingly, within a company network, rather than just groupingservers, servers and groups of server systems are organized asdistributed resources. There is an increased effort to collaborate,share data, share cycles, and improve other modes of interaction amongservers within a company network and outside the company network.Further, there is an increased effort to outsource nonessential elementsfrom one company network to that of a service provider network.Moreover, there is a movement to coordinate resource sharing betweenresources that are not subject to the same management system, but stilladdress issues of security, policy, payment, and membership. Forexample, resources on an individual's desktop are not typically subjectto the same management system as resources of a company server cluster.Even different administrative groups within a company network mayimplement distinct management systems.

The problems with decentralizing the resources available from serversand other computing systems operating on different network platforms,located in different regions, with different security protocols and eachcontrolled by a different management system, has led to the developmentof Grid technologies using open standards for operating a gridenvironment. Grid environments support the sharing and coordinated useof diverse resources in dynamic, distributed, virtual organizations. Avirtual organization is created within a grid environment when aselection of resources, from geographically distributed systems operatedby different organizations with differing policies and managementsystems, is organized to handle a job request. A grid vendor may developa grid environment to which a buyer may submit grid jobs, for example.

Grid vendors may offer to process grid jobs with different performancepromises and with different pricing policies. Even if standards, such asthose proposed by the open standards organization for Grid technologies,define standard monitoring, metering, rating, accounting, and billinginterfaces, grid vendors will still have different physical resourcesavailable to process grid jobs, and thus pricing and performance willstill vary among grid vendors. In one example, grid vendors have tomeasure the use of the grid vendor's resources by a grid job, which mayinvolve complex formulas which take into account multiple factors, inaddition to the actual use of resources. For example, a grid vendor maydedicate a particular processor resource to a particular job and chargethe grid job for the dedicated use of the processor, in addition to theactual number of processor cycles the grid job required.

While grid vendors are focused on monitoring, metering, accounting abilling for the actual usage of physical resources, at a computationalcycle level, grid clients or customers are focused on processing ofapplications and jobs at an application type level. As a result, thereis a lack of connection between the way that grid customers and gridvendors view the costs associated with grid jobs. Further, currently,each grid vendor still monitors, meters, and bills for grid jobs usingdifferent units of physical resource measurement. Thus, because of thedisconnect between the client grid job at an application level and thegrid vendor measurement of use of physical resources, it is difficultfor grid clients to compare the costs of processing grid jobs atdifferent grid vendors and to estimate future costs of submitting gridjobs to a same grid vendor.

Therefore, in view of the foregoing, it would have advantageous toprovide a method, system, and program for estimating future job costs byclassifying grid jobs in categories with client-defined applicationbased metric units, converting the grid vendor defined metric costs toperform grid jobs into the client-defined application based metric unitcosts by category of grid job, and storing the converted client-definedapplication based metric unit costs for predicting future costs of gridjobs of the same category. In particular, it would be advantageous tosubmit grid job microcosms, or smaller representative grid jobs, tomultiple grid vendors to retrieve actual costs for each category of gridjob on a smaller basis, converting the grid vendor defined metric coststo a client-defined application based metric unit cost, and comparingthe costs at the client-defined application based metric level, beforesubmitting larger grid jobs, in the future to the most cost effectivegrid vendor.

SUMMARY OF THE INVENTION

In view of the foregoing, the present invention in general provides forimproved grid computing and in particular to client-side estimation offuture grid job costs. Still more particularly, the present inventionrelates to customer estimation of future grid job costs by comparing acurrent grid job of a particular classification with a history of storedcosts for other grid jobs of that customer of that particularclassification.

In one embodiment, a grid client agent for a client system enabled tosubmit grid jobs to a grid provider that facilitates an on-demand gridenvironment, calculates a ratio of an application based metric to a gridprovider metric for processing a particular grid job. Then, the gridclient agent creates a table with an entry comparing the applicationbased metric to a cost per grid provider metric for the grid providerbased on the calculated ratio. Next, the grid client agent stores thetable with the entry. Then, responsive to detecting a next grid job, thegrid client agent estimates a cost for the grid provider to process thenext grid job based on a particular number of application based metricoperations required for the next grid job, translated by the ratio intothe grid provider metric and multiplied by the cost per grid providermetric.

To calculate the ratio, first, the grid client agent distributes a jobmicrocosm, which is a smaller representation of a particular grid job,to the grid provider for processing in the on-demand grid environment.Responsive to receiving the result of the job microcosm and a charge forprocessing the job microcosm based on a grid provider metric for thegrid provider, then grid client agent calculates the ratio of theapplication based metric to the grid provider metric and identifies thecost per grid provider metric from the charge for processing the jobmicrocosm. In addition, the grid client agent may first distribute a jobrequest with requirements for the grid microcosm to the grid providerand receive a bid from the grid provider specifying the cost and otheragreements for processing the grid microcosm. Further, in addition, thegrid client agent may also identify the cost per grid provider metricfrom a published rate by the grid provider.

When the grid client agent detects an adjusted cost per grid providermetric, whether through a pricing notification or the charges foranother grid job, the grid client agent updates the entry for the gridprovider with the adjusted cost per grid provider metric andautomatically reestimates the cost for the grid provider to process thenext grid job based on the adjusted cost per grid provider metric,without requiring a new calculation of the ratio.

The table may include additional entries from additional grid providerswho process grid microcosms of a particular job, where the grid clientagent calculates a ratio of the application based metric to each gridproviders different metric for each entry in the table. Where multiplegrid provider entries are available in the table, then the grid clientagent may estimate the cost for each grid provider to process the nextgrid job based on the ratio and cost per grid provider metric in eachentry and compare the costs which are calculated for the applicationbased metric number of operation required for the next grid job.

In addition, entries in the table are classified by category of gridjob. Thus, the grid client agent will detect the next grid job, classifythe grid job within one of the categories of grid job, and access thoseentries in the table that are also classified by the same category ofgrid job.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features believed aspect of the invention are set forth in theappended claims. The invention itself however, as well as a preferredmode of use, further objects and advantages thereof, will best beunderstood by reference to the following detailed description of anillustrative embodiment when read in conjunction with the accompanyingdrawings, wherein:

FIG. 1 depicts one embodiment of a computer system which may beimplemented in a grid environment and in which the present invention maybe implemented;

FIG. 2 is block diagram illustrating one embodiment of the general typesof components within a grid environment;

FIG. 3 is a block diagram illustrating one example of an architecturethat may be implemented in a grid environment;

FIG. 4 is a flow diagram depicting a job request submitted by a clientsystem to a grid provider and the resulting bid for that job requestreturned by the grid provider;

FIG. 5 is a flow diagram illustrating a job submitting by a clientsystem to a grid provider and the results of the job returned by thegrid provider;

FIG. 6 is a flow diagram depicting a process for submitting a microcosmof a grid job to multiple grid providers to calculate a relative costper client-defined application metric;

FIG. 7 is a block diagram depicting a grid client agent for estimatingfuture grid job costs by comparing a current grid job of a particularclassification with a history of stored costs for other grid jobs ofthat particular classification;

FIG. 8 is a table illustrating examples of client-defined applicationmetric based costs compared with grid provider metric based costs forcategories of grid jobs; and

FIG. 9 is a high level logic flowchart depicting a process and programfor a grid client agent to estimate future costs of grid jobs of aclient-defined classification in accordance with the method, system, andprogram of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Referring now to the drawings and in particular to FIG. 1, there isdepicted one embodiment of a computer system which may be implemented ina grid environment and in which the present invention may beimplemented. As will be further described, the grid environment includesmultiple computer systems managed to provide resources. Additionally, aswill be further described, the present invention may be executed in avariety of computer systems, including a variety of computing systems,mobile systems, and electronic devices operating under a number ofdifferent operating systems managed within a grid environment.

In one embodiment, computer system 100 includes a bus 122 or otherdevice for communicating information within computer system 100, and atleast one processing device such as processor 112, coupled to bus 122for processing information. Bus 122 may include low-latency and higherlatency paths connected by bridges and adapters and controlled withincomputer system 100 by multiple bus controllers. When implemented as aserver system, computer system 100 typically includes multipleprocessors designed to improve network servicing power.

Processor 112 may be a general-purpose processor such as IBM's PowerPC(PowerPC is a registered trademark of International Business MachinesCorporation) processor that, during normal operation, processes dataunder the control of operating system and application softwareaccessible from a dynamic storage device such as random access memory(RAM) 114 and a static storage device such as Read Only Memory (ROM)116. The operating system may provide a graphical user interface (GUI)to the user. In one embodiment, application software contains machineexecutable instructions that when executed on processor 112 carry outthe operations depicted in the flowchart of FIG. 9 and others operationsdescribed herein. Alternatively, the steps of the present inventionmight be performed by specific hardware components that containhardwired logic for performing the steps, or by any combination ofprogrammed computer components and custom hardware components.

The present invention may be provided as a computer program product,included on a machine-readable medium having stored thereon the machineexecutable instructions used to program computer system 100 to perform aprocess according to the present invention. The term “machine-readablemedium” as used herein includes any medium that participates inproviding instructions to processor 112 or other components of computersystem 100 for execution. Such a medium may take many forms including,but not limited to, non-volatile media, volatile media, and transmissionmedia. Common forms of non-volatile media include, for example, a floppydisk, a flexible disk, a hard disk, magnetic tape or any other magneticmedium, a compact disc ROM (CD-ROM) or any other optical medium, punchcards or any other physical medium with patterns of holes, aprogrammable ROM (PROM), an erasable PROM (EPROM), electrically EPROM(EEPROM), a flash memory, any other memory chip or cartridge, or anyother medium from which computer system 100 can read and which issuitable for storing instructions. In the present embodiment, an exampleof a non-volatile medium is mass storage device 118 which as depicted isan internal component of computer system 100, but will be understood toalso be provided by an external device. Volatile media include dynamicmemory such as RAM 114. Transmission media include coaxial cables,copper wire or fiber optics, including the wires that comprise bus 122.Transmission media can also take the form of acoustic or light waves,such as those generated during radio frequency or infrared datacommunications.

Moreover, the present invention may be downloaded as a computer programproduct, wherein the program instructions may be transferred from aremote virtual resource, such as a virtual resource 160, to requestingcomputer system 100 by way of data signals embodied in a carrier wave orother propagation medium via a network link 134 (e.g. a modem or networkconnection) to a communications interface 132 coupled to bus 122.Virtual resource 160 may include a virtual representation of theresources accessible from a single system or systems, wherein multiplesystems may each be considered discrete sets of resources operating onindependent platforms, but coordinated as a virtual resource by a gridmanager. Communications interface 132 provides a two-way datacommunications coupling to network link 134 that may be connected, forexample, to a local area network (LAN), wide area network (WAN), or anInternet Service Provider (ISP) that provide access to network 102. Inparticular, network link 134 may provide wired and/or wireless networkcommunications to one or more networks, such as network 102, throughwhich use of virtual resources, such as virtual resource 160, isaccessible as provided within a grid environment 150. Grid environment150 may be part of multiple types of networks, including a peer-to-peernetwork, or may be part of a single computer system, such as computersystem 100.

As one example, network 102 may refer to the worldwide collection ofnetworks and gateways that use a particular protocol, such asTransmission Control Protocol (TCP) and Internet Protocol (IP), tocommunicate with one another. Network 102 uses electrical,electromagnetic, or optical signals that carry digital data streams. Thesignals through the various networks and the signals on network link 134and through communication interface 132, which carry the digital data toand from computer system 100, are exemplary forms of carrier wavestransporting the information. It will be understood that alternate typesof networks, combinations of networks, and infrastructures of networksmay be implemented.

When implemented as a server system, computer system 100 typicallyincludes multiple communication interfaces accessible via multipleperipheral component interconnect (PCI) bus bridges connected to aninput/output controller. In this manner, computer system 100 allowsconnections to multiple network computers.

Additionally, although not depicted, multiple peripheral components andinternal/external devices may be added to computer system 100, connectedto multiple controllers, adapters, and expansion slots coupled to one ofthe multiple levels of bus 122. For example, a display device, audiodevice, keyboard, or cursor control device may be added as a peripheralcomponent.

Those of ordinary skill in the art will appreciate that the hardwaredepicted in FIG. 1 may vary. Furthermore, those of ordinary skill in theart will appreciate that the depicted example is not meant to implyarchitectural limitations with respect to the present invention.

With reference now to FIG. 2, a block diagram illustrates one embodimentof the general types of components within a grid environment. In thepresent example, the components of a grid environment 150 include aclient system 200 interfacing with a grid management system 240 whichinterfaces with server clusters 222, servers 224, workstations anddesktops 226, data storage systems 228, and networks 230. For purposesof illustration, the network locations and types of networks connectingthe components within grid environment 150 are not depicted. It will beunderstood, however, that the components within grid environment 150 mayreside atop a network infrastructure architecture that may beimplemented with multiple types of networks overlapping one another.Network infrastructure may range from multiple large enterprise systemsto a peer-to-peer system to a single computer system. Further, it willbe understood that the components within grid environment 150 are merelyrepresentations of the types of components within a grid environment. Agrid environment may simply be encompassed in a single computer systemor may encompass multiple enterprises of systems.

It will be understood that grid environment 150 may be provided by agrid vendor or provider, where a cost for use of resources within gridenvironment 150 may be calculated based on the amount of time requiredfor a grid job to execute or the actual amount of resources used, forexample. In addition, it will be understood that grid environment 150may include grid resources supplied by a single grid vendor, such as aparticular business enterprise, or multiple vendors, where each vendorcontinues to monitor and manage the vendor's group of resources, butgrid management system 240 is able to monitor unintended changes acrossall the resources, regardless of which vendors provide which resources.Further, it will be understood that although resource discoverymechanisms for discovering available grid resources are not depicted,client system 200 or grid management system 240 may discover gridresources advertised from local and global directories available withinand outside of grid environment 150.

The central goal of a grid environment, such as grid environment 150 isorganization and delivery of resources from multiple discrete systemsviewed as virtual resource 160. Client system 200, server clusters 222,servers 224, workstations and desktops 226, data storage systems 228,networks 230 and the systems creating grid management system 240 may beheterogeneous and regionally distributed with independent managementsystems, but enabled to exchange information, resources, and servicesthrough a grid infrastructure enabled by grid management system 240.Further, server clusters 222, servers 224, workstations and desktops226, data storage systems 228, and networks 230 may be geographicallydistributed across countries and continents or locally accessible to oneanother.

In the example, client system 200 interfaces with grid management system240. Client system 200 may represent any computing system sendingrequests to grid management system 240. In particular, client system 200may send virtual job requests (or requests for a quote (RFQs) and jobsto grid management system 240. Further, while in the present embodimentclient system 200 is depicted as accessing grid environment 150 with arequest, in alternate embodiments client system 200 may also operatewithin grid environment 150.

While the systems within virtual resource 160 are depicted in parallel,in reality, the systems may be part of a hierarchy of systems where somesystems within virtual resource 160 may be local to client system 200,while other systems require access to external networks. Additionally,it is important to note, that systems depicted within virtual resources160 may be physically encompassed within client system 200.

To implement grid environment 150, grid management system 240facilitates grid services. Grid services may be designed according tomultiple architectures, including, but not limited to, the Open GridServices Architecture (OGSA). In particular, grid management system 240refers to the management environment which creates a grid by linkingcomputing systems into a heterogeneous network environment characterizedby sharing of resources through grid services.

In particular, as will be described with reference to FIGS. 4-6, gridmanagement system 240 may include grid services that enable automatedresponses to bid requests and automated monitoring and metering of gridjobs. In addition, grid management system 240 may include additionalservices for automating functions performed within grid environment 150.

According to an advantage of the invention, client system 200 includes agrid client agent for estimating future costs of grid jobs. As will bedescribed with reference to FIG. 7, the grid client agent estimatesfuture costs of grid jobs by classifying grid jobs into categories,storing previously charged costs for each category of grid job in atable, and accessing the table to predict future costs of grid jobs bycategory.

Referring now to FIG. 3, a block diagram illustrates one example of anarchitecture that may be implemented in a grid environment. As depicted,an architecture 300 includes multiple layers of functionality. As willbe further described, the present invention is a process which may beimplemented in one or more layers of an architecture, such asarchitecture 300, which is implemented in a grid environment, such asthe grid environment described in FIG. 2. It is important to note thatarchitecture 300 is just one example of an architecture that may beimplemented in a grid environment and in which the present invention maybe implemented. Further, it is important to note that multiplearchitectures may be implemented within a grid environment.

Within the layers of architecture 300, first, a physical and logicalresources layer 330 organizes the resources of the systems in the grid.Physical resources include, but are not limited to, servers, storagemedia, and networks. The logical resources virtualize and aggregate thephysical layer into usable resources such as operating systems,processing power, memory, I/O processing, file systems, databasemanagers, directories, memory managers, and other resources.

Next, a web services layer 320 provides an interface between gridservices 310 and physical and logical resources 330. Web services layer320 implements service interfaces including, but not limited to, WebServices Description Language (WSDL), Simple Object Access Protocol(SOAP), and eXtensible mark-up language (XML) executing atop an InternetProtocol (IP) or other network transport layer. Further, the Open GridServices Infrastructure (OSGI) standard 322 builds on top of current webservices 320 by extending web services 320 to provide capabilities fordynamic and manageable Web services required to model the resources ofthe grid. In particular, by implementing OGSI standard 322 with webservices 320, grid services 310 designed using OGSA are interoperable.In alternate embodiments, other infrastructures or additionalinfrastructures may be implemented a top web services layer 320.

Grid services layer 310 includes multiple services, the combination ofwhich may implement grid management system 240. For example, gridservices layer 310 may include grid services designed using OGSA, suchthat a uniform standard is implemented in creating grid services.Alternatively, grid services may be designed under multiplearchitectures. Grid services can be grouped into four main functions. Itwill be understood, however, that other functions may be performed bygrid services.

First, a resource management service 302 manages the use of the physicaland logical resources. Resources may include, but are not limited to,processing resources, memory resources, and storage resources.Management of these resources includes scheduling jobs, distributingjobs, and managing the retrieval of the results for jobs. Resourcemanagement service 302 monitors resource loads and distributes jobs toless busy parts of the grid to balance resource loads and absorbunexpected peaks of activity. In particular, a user may specifypreferred performance levels so that resource management service 302distributes jobs to maintain the preferred performance levels within thegrid.

Second, information services 304 manages the information transfer andcommunication between computing systems within the grid. Since multiplecommunication protocols may be implemented, information services 304manages communications across multiple networks utilizing multiple typesof communication protocols.

Third, a data management service 306 manages data transfer and storagewithin the grid. In particular, data management service 306 may movedata to nodes within the grid where a job requiring the data willexecute. A particular type of transfer protocol, such as Grid FileTransfer Protocol (GridFTP), may be implemented.

Finally, a security service 308 applies a security protocol for securityat the connection layers of each of the systems operating within thegrid. Security service 308 may implement security protocols, such asOpen Secure Socket Layers (SSL), to provide secure transmissions.Further, security service 308 may provide a single sign-on mechanism, sothat once a user is authenticated, a proxy certificate is created andused when performing actions within the grid for the user.

Multiple services may work together to provide several key functions ofa grid computing system. In a first example, computational tasks aredistributed within a grid. Data management service 306 may divide up acomputation task into separate grid services requests of packets of datathat are then distributed by and managed by resource management service302. The results are collected and consolidated by data managementsystem 306. In a second example, the storage resources across multiplecomputing systems in the grid are viewed as a single virtual datastorage system managed by data management service 306 and monitored byresource management service 302.

An applications layer 340 includes applications that use one or more ofthe grid services available in grid services layer 310. Advantageously,applications interface with the physical and logical resources 330 viagrid services layer 310 and web services 320, such that multipleheterogeneous systems can interact and interoperate.

With reference now to FIG. 4, there is depicted a flow diagram of a jobrequest submitted by a client system to a grid provider and theresulting bid for that job request returned by the grid provider. Asillustrated, client system 200 submits a job request 402 to a gridprovider. In particular, each grid provider may implement a gridmanagement system for managing the bidding process on job requests andfor managing the resulting flow of the job through a selection of gridresources managed by the grid provider through the grid managementsystem.

In the example, the grid management system for a grid provider includesa grid provider bid request portal 404 at which job requests arereceived and queued. Grid provider bid request portal directs each jobrequest to a workload calculator 408 which calculates the workloadrequirements of job request 402. In particular, workload requirementsmay include, for example, an estimation of the computational cycles thata job will require and the type of hardware and software platformsrequired. Workload calculator 408 distributes the workload calculationsas workload data 412 to a bid formalizer 418 and as workload data 410 toa cost calculator 414. Cost calculator 414 uses the workloadcalculation, job request requirements, and current and estimated costsfor use of resources to estimate a cost for processing the grid jobspecified in job request 402. Cost calculator 414 returns cost data 416to bid formalizer 418. Bid formalizer 418 gathers workload data 412 andcost data 416 into a bid 420 which is returned from the grid provider toclient system 200. Bid 420 may agree to perform the grid job exactly asrequested or may include exceptions, exclusions, and other variationsfrom the specification in job request 402. In addition, bid 420 may beviewed as a service level agreement, specifying a performance standardwhich the grid provider agrees to if the grid job is later submitted tothe grid provider. Further, bid formalizer 418 may create a bid based ona pricing contract agreement reached between the grid client and theprovider before or after the bid placement.

Referring now to FIG. 5, there is depicted a flow diagram of a jobsubmitting by a client system to a grid provider and the results of thejob returned by the grid provider. As illustrated, client system 200submits job 502 to a grid provider. In particular, as described withreference to FIG. 4, each grid provider may implement a grid managementsystem for managing the flow of a grid job by selecting grid resourcesfor a job to meet a performance requirements, monitoring the progress ofjobs and adjusting grid resources if needed to meet performancerequirements, and managing the return of results to the client system.

In the example, the grid management system for a grid provider includesa job queue 504 that receives job 502 and holds job 502 until gridscheduler 506 can schedule and dispatch job 502 to grid resources. Inparticular, grid scheduler 502 accesses service level agreement (SLA)508, which includes the performance requirements for job 502, based on abid placed by the grid provider for the specific job or an agreement forjob performance requirements for jobs received from a particular clientsystem, for example. Grid scheduler 506 accesses the grid resourcesrequired to handle job 502, for example server A 516, server B 518, andserver N 520. Although not depicted, grid scheduler 506 may access agrid manager and other components of the grid management system thatbuild the required resources for a grid job, access resources from othergrid environments, and sell-off grid jobs if necessary to other gridproviders.

In the example, grid scheduler 506 divides job 502 into job parts 510,512, and 514 that are distributed to server A 516, server B 518, andserver N 520, respectively. A job results manager 528 collects results522, 524, and 526 from serve A 516, server B 518, and server N 520,respectively. Job results manager 528 returns complete results 530 toclient system 200. In addition, job results manager 528 updates anaccounting manager 532 when the job is complete. Accounting manager 532communicates with a workload manager (not depicted) that monitors theuse of server A 516, server B 518, and server N 520 by job 502 tocalculate the total workload of job 502 and the total cost of job 502.In particular SLA 508 may specify factors that control the total cost ofjob 502, such as a maximum cost, a fixed cost, a sliding cost scale ifperformance requirements are not met, and other pricing adjustmentfactors.

With reference now to FIG. 6, there is depicted a flow diagram of aprocess for submitting a grid job microcosms to multiple grid providersto calculate a relative cost per client-defined application metric. Inthe example, a client system 200 apportions a grid job into microcosms,which are small, representative jobs of the larger grid job that needsto be submitted to a grid provider. In the example, client system 200submits grid job microcosm 602 and grid job microcosm 612 to gridproviders 604 and 614, respectively. In one embodiment, client system200 has already submitted a job request for the grid job microcosm togrid providers 604 and 614, as described with reference to FIG. 4,however, in an alternate embodiment, client system 200 may submit thegrid job microcosms to grid providers 604 and 614 with pricing andperformance expectations based on published rates or verbally agreed torates, for example.

Each of grid providers 604 and 614 process gird job microcosms 602 and612 and return results 606 and 616 the same manner as described withreference to a grid provider processing a grid job in FIG. 5. Clientsystem 200, as will be further described with reference to FIGS. 7, 8,and 9, retrieves the results and costs for each of grid job microcosms602 and 604 and calculates a translation value for each grid providerbased on a client-defined application metric to grid provider metricratio. Then, client system 200 calculates an estimated cost for the fullgrid job according to the number of client-defined application metricoperations required for the full grid job adjusted by the translationvalue, compares the costs estimated for each grid provider, and selectsthe most cost effective provider.

According to an advantage, by sampling the actual performance and costfor each provider and translating the cost into a client-definedapplication metric basis, client system 200 can compare the actual costfor performance, rather than the promised cost for performance, onclient-defined application metric basis, before sending a large grid jobor multiple large grid jobs. In the example, after sampling the resultsand cost for each of grid job microcosms 602 and 612, client system 200selects to send full grid job 620, of which grid job microcosm 602 and612 are representative sets, to grid provider 604. Grid provider 604processes full grid job 620, as described with reference to FIG. 5, andreturns result 624 to client system 200.

Referring now to FIG. 7, a block diagram depicts a grid client agent forestimating future grid job costs by comparing a current grid job of aparticular classification with a history of stored costs for other gridjobs of that particular classification. In the example, grid clientagent 700 describes software modules operating within client system 200or another system that enables client system 200.

A job microcosm controller 702 controls the process, as described withreference to FIG. 6, for submitting job microcosms of a particular gridjob to multiple grid providers.

In particular, job microcosm controller 702 may first query multiplegrid providers with a job request for the job microcosm. In addition toquerying grid providers with job requests as described with reference toFIG. 6, additional rate information may be acquired verbally, inwriting, via email or through some other electronic interchange method.Job microcosm controller 702 may store the bids or rate quotes,designating a price per computational cycle. In one example, a firstprovider returns a price per hour, a second provider returns a price perfloating point operation and a third provider returns a price based onthe complex formula of the provider's grid environment.

Next, once job microcosm controller 702 acquires bids and rate quotesfrom multiple grid providers, job microcosm controller 702 submits jobmicrocosms, which are small jobs representative of larger grid jobs tobe submitted, to a selection of the multiple grid providers. In oneexample, if a corporation needs an average of 20,000,000 records mergedeach night, then the job microcosm distributed to each of the selectedgrid providers may include 1% of these records. In another example, aclient does not send a portion of the actual grid job, but insteadsubmits a job microcosm of an analogous job with tester data.

When job microcosm controller 702 receives the computational results ofthe job microcosms are received, the charges from each grid provider foreach job microcosm are already received. Job microcosm controller 702also detects the time taken, once the job microcosm was submitted to agrid provider, for the grid provider to return a result. In the example,the first provider takes five minutes to return a result and charges$2.20, the second provider takes one minute to return a result andcharges $1.74 and the third provider takes two minutes to return aresult and charges $3.40. In particular, it will be understood that jobmicrocosm controller 702 may receive the charges from each grid providerthrough multiple communication media, including a separate transmissionfrom the grid provider to client system 200, an email communication, anembedded accounting token digitally signed and returned to client system200 with a transaction receipt.

Once all the costs per grid microcosm are received, a cost comparator706 compares the actual costs by grid provider for performing aparticular category of grid job. In particular, cost comparator 706calculates a cost by client-defined application metric for each grid jobmicrocosm. In particular, each grid provider submits provider-definedmetric costs, such as cost per hour or cost per provider-based complexformula. The client, however, defines grid jobs at an application levelgranularity. For example, a client defined application metric is a costper record merge. Once cost comparator 706 calculates a client-definedapplication metric to grid-provider metric ratio, then cost comparator706 can translate the number of client-defined application metricoperations required for a full job into a price using the client-definedapplication metric to grid-metric ratio as a translation value. Costcomparator 706 determines the most cost effective grid provider andtriggers submission of the remainder of the grid job or the actual gridjob to the most cost effective grid provider.

In addition cost comparator 706 calculates the client-definedapplication metric to grid-provide metric cost ratio for cost tables 710and stores the cost by client-defined application metric in cost tables710. As illustrated with reference to FIG. 8, in particular, cost tables710 includes a column for a client-defined application metric 804. Eachclient-defined application metric, although not depicted, is associatedwith a client-defined category. Client-defined categories are familiesof grid jobs classified at an application level for performing similartypes of operations. Grid job classifier 704 may define categories orcategories may be entered and particular grid jobs classified. In theexample carried through in FIG. 8, the client-defined categories are“batch merges” with a client-defined application metric of “per merge”and “meteorological operations” with a client-defined application metricof “per model iteration”. These client-defined categories may apply, forexample, where the client is a university that performs only twocomputational functions. First, the university system performs payrollfunctions, which requires database batch merges. Second, the universitysystem performs computationally-intensive research work which requiresgreater floating point operations. It will be understood that additionalcategories may be added to classify other types of operations typicallyperformed by a client in need of the computational power and otherresources provided by on-demand grid providers.

Cost tables 710 includes a second column for a provider identifier 806.In the example, values listed under provider identifiers 806 are “acmegrid”, “wiley grid”, and “coyote grid”. It will be understood that othertypes of provider identifiers may be implemented, including an addressand other indicia of a grid provider.

Cost tables 710 includes a third column for a grid-provider metric 808.In the example, the values listed under grid-provider metric 808 are“hourly charge”, “proprietary composite charge”, and “million floatingpoint (MFP) operations charge”. It will be understood that additionaltypes of grid-provider metric values may be defined by grid providers.Further, it will be understood that grid providers may designate agrid-provider metric when bidding on a job request or with the chargesreturned for processing a grid microcosm. In addition, a “proprietarycomposite charge” refers to a charge calculated by the grid providerbased on multiple factors, including for example, the data volume movedacross the network, jobs submitted to the processor run queues, andbytes written to and read from a grid provider's own storage system.

A fourth column in cost tables 710 includes a translation value 810which represents the ratio of the client-defined application metric tothe grid-provider metric. In the example, values listed undertranslation value 810 include “3,000,000 merges per hour”, “600 mergesper composite unit”, and “2000 merges per MFP operations”. Inparticular, the translation values are calculated by cost comparator 706and represent the number of client-defined application metricsaccomplished per grid-provider metric ratio. As previously described,translation values may be calculated based on grid microcosm. In otherembodiments, however, translation values may also be calculated andupdated based on a full job submission.

Finally, the fifth column in cost tables 710 includes an offered pricingper grid-provider metric 812. In the example, the values listed underoffered pricing per grid-provider metric 812 include “$40 per hour”,“$0.02 per composite unit”, and “$0.08 per MFP”. In an alternateembodiment, historical pricing ranges may be given by provider, as wellas the most recent price by provider.

According to an advantage of the invention, when a client wants toestimate a cost of a new grid job, grid job classifier 704 classifiesthe grid job and future cost estimator 708 searches cost tables 710 forclient-defined application metric based costs for that category of gridjob. Then, based on the client-defined application metric requirementsof the new grid job, future cost estimator 708 estimates the cost forthe new grid job according to grid provider. In one example, based onthe values illustrated in cost tables 710, a new grid job requiring3,000,000 batch merges would cost $40 on “acme grid” (3,000,000 mergesper hour/$40 per hour), $100 on “wiley grid” (600 merges per compositeunit/$0.02 per composite unit), and $120 on “coyote grid” (2000 mergesper MFP operations/$0.08 per MFP).

Further, according to the advantage, if future cost estimator 708searches cost tables 710 for client-defined application metric basedcosts for that category of grid job and none or available or the costsare out of date, then future cost estimator 708 initiates job microcosmcontroller 702 to determine current costs for microcosms of theparticular classification category of grid job.

In addition, it is important to note that as pricing per grid-providermetric changes over time or in response to market conditions, the newcost can be inserted into the offered pricing per grid-provider metric812 column in cost tables 710 and prices estimates for jobs classifiedwithin the categories updated, without changing the translation values.For example, if the “acme grid” price increases to $55 per hour, the“wiley grid” price drops to $0.013 per composite unit, and the “coyotegrid” price drops to “$0.05 per MFP”, the client-defined applicationmetric to grid-provider metric ratio listed under the translation value810 column does not change, so future cost estimator 708 can stillestimate the future cost to complete a 5,000,000 batch merge jobtomorrow based on the updated prices, e.g. $91.67 on “acme grid”(3,000,000 merges per hour/$55 per hour), $108.33 on “wiley grid” (600merger per composite unit/$0.013 per composite unit) and $125 on the“coyote grid” (2000 mergers per MFP operation/$0.05 per MFP).

Referring now to FIG. 9, there is depicted a high level logic flowchartof a process and program for a grid client agent to estimate futurecosts of grid jobs of a client-defined classification in accordance withthe method, system, and program of the present invention. Asillustrated, the process starts at block 900 and thereafter proceeds toblock 902. Block 902 depicts a determination of classifying a grid jobaccording to nature into a client-defined category. Next, block 904illustrates a determination whether grid jobs of the same category arealready priced in the cost table. If grid jobs of the same category arealready priced in the cost table, then the process passes to block 906.Block 906 depicts predicting the cost of the current grid job based onthe previously stored pricing in the cost table for the particularcategory of grid job, and the process ends. In particular, the costtable is cost table 710, which includes the translation value. Thus,predicting the cost of a current grid job requires determining thenumber of client-defined application metric operations required andconverting the number of client-defined application metric operations toa grid-provider metric based cost through the translation value of thecategory of grid job.

Returning to block 904, if there are not already grid jobs of the samecategory priced in the cost table, then the process passes to block 908.In addition, although not depicted, at block 904 a determination mayalso be made that even though there grid jobs of the same categoryalready priced in the cost table, that the pricing is outdated, and theprocess passes to block 908.

Block 908 depicts creating a grid job request for a small part of thegrid job. Next, block 910 illustrates distributing the grid job requestto multiple grid providers. Thereafter, block 912 depicts adetermination whether the grid client agent receives grid bids for thesmall part of the grid job. As depicted, if no bids are yet received,the process iterates at block 912, however, if no bids are receivedafter a period of time, then the job request may be adjusted andresubmitted to the multiple grid providers. Once bids are received, thenthe process passes to block 914.

Block 914 depicts selecting those grid job providers whose bids meet thebid request requirements. Next, block 916 illustrates distributing smallparts of the grid job to the selection of the grid job providers.Thereafter, block 918 depicts a determination whether the grid clientagent receives all the results of the grid job processing with costs. Asillustrated, if the results are not yet received, the process iteratesat block 918, however, if not all results are received within theexpected period of time for response, then those grid providers notreturning a results may be queried. Once the results are retrieved, thenthe process passes to block 920.

Block 920 depicts calculating the client-defined application metric togrid provider metric ratio for each grid provider. Next, block 921illustrates calculating the cost per grid provider using the ratio basedon the number of client-defined application metric operations requiredfor the large grid job. Next, block 922 illustrates comparing the actualcosts per grid provider. Thereafter, block 924 depicts distributing theremainder of the grid job to the projected least expensive grid providerbased on client-defined application metrics. Then, block 926 illustratesstoring the ratio and grid provider costs in the cost table according tothe client-defined classification of the grid job, and the process ends.

While the invention has been particularly shown and described withreference to a preferred embodiment, it will be understood by thoseskilled in the art that various changes in form and detail may be madetherein without departing from the spirit and scope of the invention.

1. A system for estimating a future cost of a grid job, said system comprising: a client system communicatively connected to a network, wherein said client system is enabled to submit a particular grid job to a grid provider that facilitates a grid environment for processing said particular grid job; said client system further comprising: means for calculating a ratio of an application based metric to a grid provider metric for said grid provider processing said particular grid job by: distributing a job microcosm of said particular grid job from said client system to said grid provider via a network for processing in said grid environment; and responsive to receiving a result of said job microcosm and a charge for processing said job microcosm based on a grid provider metric from said grid provider, calculating at said client system said ratio of said application based metric to said grid provider metric and identifying said cost per grid provider metric from said charge for processing said job microcosm; means for creating a table with an entry comparing said application based metric to a cost per grid provider metric for said grid provider with said ratio; means for storing said table with said entry for said particular grid job; means, responsive to said client system detecting a next grid job classified by said at least one application metric, for accessing said entry comprising said application based metric to said cost per grid provider metric for said grid provider with said radio; and means for calculating an estimated cost for said grid provider to process a next grid job based on a particular number of application based metric operations required for said next grid job translated by said ratio into said grid provider metric and multiplied by said cost per grid provider metric.
 2. The system according to claim 1 for estimating a future cost of a grid job, said client system further comprising: means for distributing a job request specifying requirements for processing said job microcosm to said grid provider; and means, responsive to receiving a bid for said grid provider to process said job microcosm according to requirements, for distributing said job microcosm to said grid provider for processing.
 3. The system according to claim 1 for estimating a future cost of a grid job, wherein said means for creating a table with an entry comparing said application based metric to a cost per grid provider metric for said grid provider with said ratio, further comprises: means for accessing said cost per grid provider metric from at least one from among charges received from said grid provider for processing said particular grid job and a published rate per grid provider metric.
 4. The system according to claim 1 for estimating a future cost of a grid job, said client system further comprising: means, responsive to detecting at said client system an adjusted cost per grid provider metric for said grid provider, for updating said cost per grid provider metric in said table; and means for automatically reestimating said cost for said grid provider to process said next grid job based on said particular number of application based metric operations required for said next job translated by said ratio into said grid provider metric and multiplied by said adjusted cost per grid provider metric.
 5. The system according to claim 1 for estimating a future cost of a grid job, said client system further comprising: means for submitting a plurality of job microcosms of said particular grid job to a plurality of grid providers; means, responsive to receiving a plurality of results and a plurality of charges for processing said plurality of grid microcosms from each of said plurality of grid providers, for calculating a plurality of ratios of said application based metric to each of said plurality of grid providers metrics for processing said particular grid job; means for creating said table with a plurality of entries, where each entry compares said application based metric to a cost for one of said plurality of grid providers per each of said plurality of grid provider metrics a particular ration for said one of said plurality of grid providers; means for storing each of said plurality of entries in said table; means for estimating a plurality of costs based on said application based metric for each of said plurality of grid providers to process said next job; and means for comparing said plurality of costs to identify a least expensive grid provider from among said plurality of grid providers.
 6. The system according to claim 1 for estimating a future cost of a grid job, said clients system further comprising: means for classifying said particular grid job into a particular category from among a plurality of categories of grid jobs, wherein said particular category comprises grid jobs with operations measurable by said application based metric; means for storing said entry for said particular grid job according to said particular category; means, responsive to detecting a next grid job, for classifying said next grid job according to said particular category of grid job; means for searching said table for said entry associated with said same particular category; and means for estimating a cost for said grid provider to process said next grid job in said same particular category as said particular grid job based on said entry calculated based on said particular grid job.
 7. A program for estimating a future cost of a grid job, said program embodied in a volatile or non-volatile computer-readable medium, said program comprising computer-executable instructions which cause a computer to perform the steps of: calculating a ratio of an application based metric to a grid provider metric for processing said particular grid job by a particular grid provider, wherein said grid provider facilitates a grid environment enabled to process said particular grid job by: distributing a job microcosm of said particular grid job from said client system to said grid provider via a network for processing in said grid environment; and responsive to receiving a result of said job microcosm and a charge for processing said job microcosm based on a grid provider metric from said grid provider, calculating at said client system said ratio of said application based metric to said grid provider metric and identifying said cost per grid provider metric from said charge for processing said job microcosm; creating a table with an entry comparing said application based metric to a cost per grid provider metric for said grid provider with said ratio; storing said table with said entry for said particular grid job; responsive to detecting a next grid job classified by said at least one application metric, accessing said entry comprising said application based metric to said cost per grid provider metric for said grid provider with said radio; and calculating an estimated cost for said grid provider to process said next grid job based on a particular number of application based metric operations required for said next grid job translated by said ratio into said grid provider metric and multiplied by said cost per grid provider metric.
 8. The program according to claim 7 for estimating a future cost of a grid job, further comprising the steps of: distributing a job request specifying requirements for processing said job microcosm to said grid provider; and responsive to receiving a bid for said grid provider to process said job microcosm according to requirements, distributing said job microcosm to said grid provider for processing.
 9. The program according to claim 7 for estimating a future cost of a grid job, wherein said step of creating a table with an entry comparing said application based metric to a cost per grid provider metric for said grid provider with said ratio, further comprises the step of: accessing said cost per grid provider metric from at least one from among charges received from said grid provider for processing said particular grid job and a published rate per grid provider metric.
 10. The program according to claim 7 for estimating a future cost of a grid job, further comprising the steps of: responsive to detecting an adjusted cost per grid provider metric for said grid provider, updating said cost per grid provider metric in said table; and automatically reestimating said cost for said grid provider to process said next grid job based on said particular number of application based metric operations required for said next job translated by said ratio into said grid provider metric and multiplied by said adjusted cost per grid provider metric.
 11. The program according to claim 7 for estimating a future cost of a grid job, further comprising the steps of: classifying said particular grid job into a particular category from among a plurality of categories of grid jobs, wherein said particular category comprises grid jobs with operations measurable by said application based metric; storing said entry for said particular grid job according to said particular category; responsive to detecting a next grid job, classifying said next grid job according to said particular category of grid job; searching said table for said entry associated with said same particular category; and estimating a cost for said grid provider to process said next grid job in said same particular category as said particular grid job based on said entry calculated based on said particular grid job. 